Add file-level user metadata by mprammer · Pull Request #8740 · vortex-data/vortex

mprammer · 2026-07-13T16:10:40Z

Adds optional file-level metadata to the Vortex file format: a set of string-keyed, opaque byte segments referenced from the postscript, for consumers that need to attach identity or annotation to a file (Iceberg field IDs, an Arrow-metadata round-trip, provenance). Keys and their segment locators live in the postscript and are read at open; the opaque values load only when a reader opts in with include_metadata, resolved one locator at a time through the file's existing segment source — served from the initial footer read when they fall inside it, otherwise a targeted read. A default open never materializes them and makes no metadata-driven read. The DType, its FlatBuffers and protobuf serialization, and the scan path are unchanged; the wire change is a single additive Postscript field, so old readers skip it and the file version stays 1.

This revives #7954 (the original file-metadata-segments work) rebased onto develop, with the read path reshaped so metadata is never folded into the footer's contiguous tail read and never keeps that buffer alive: values are resolved per-locator and copied out. Footer carries only the locators, so a cached footer is identical whether or not the opener asked for metadata, and one file's metadata can't surface for another through the multi-file cache. Parsing a segment alignment from an untrusted postscript now returns an error rather than panicking on an out-of-range exponent (the TUI inspector included). The public read API is additive — VortexFile gains metadata accessors and VortexOpenOptions an include_metadata opt-in — and Footer::new is unchanged; the generated Postscript FlatBuffer gains a field, as any additive schema change does.

🤖 Generated with Claude Code

Adds optional file-level metadata to the Vortex file format: string-keyed, opaque byte segments referenced from the postscript. Keys and locators live in the postscript (read at open); values load only on `include_metadata`, resolved per-locator through the segment source (from the initial footer read when covered, else a targeted read) and copied out. A default open materializes nothing. `Footer` carries only locators, so a cached footer is identical regardless of the opener's flag and one file's metadata can't surface for another via the multi-file cache. `DType`, its FlatBuffers/protobuf serialization, and the scan path are untouched; the wire change is a single additive `Postscript` field (file version stays 1). `Alignment::from_exponent` on untrusted postscript input now returns an error instead of panicking (the TUI inspector included). Revives Nicholas Gates' file-metadata-segments work (#7954), rebased onto develop and reshaped so metadata never widens the footer read or pins its backing buffer. cargo build + fmt + clippy clean; vortex-file 100/100, vortex-buffer 822/822. Co-Authored-By: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: mprammer <martin@spiraldb.com>

codspeed-hq · 2026-07-13T16:15:53Z

Merging this PR will not alter performance

✅ 1847 untouched benchmarks
⏩ 46 skipped benchmarks¹

_{Comparing mp/file-metadata (6f487e2) with develop (037b1a8)}

46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Move user metadata out of the postscript. Instead of per-key [PostscriptMetadata] locators embedded in the postscript, the postscript now carries one optional locator to a FileMetadata segment (keyed opaque values) that may live anywhere in the file. Keys no longer pressure the postscript, so the count/length limits are dropped. A default open reads no metadata; include_metadata resolves the single segment in one read. Addresses review feedback on the original approach. Co-Authored-By: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: mprammer <martin@spiraldb.com>

robert3005 · 2026-07-20T10:35:17Z

ok @mprammer sorry for back and forth. I think I remembered wrongly what this pr originally did. We can put the extra 512 bytes in the postscript for the keys. We should make sure 32 bytes covers most of the query engine keys

This reverts the single-segment rework (a9c2230), restoring the original keys-in-postscript approach per review consensus. Re-applies the CI lint fixes that the rework had carried (alignment cast, parking_lot::Mutex, to_vec) so the restored design stays green. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: mprammer <martin@spiraldb.com>

Signed-off-by: mprammer <martin@spiraldb.com> # Conflicts: # vortex-file/src/footer/deserializer.rs # vortex-file/src/footer/mod.rs

github-actions · 2026-07-20T20:48:33Z

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	`6f487e2`		1	Explore Profiling Data

Previous Runs (2)

Status	Commit	Job	Attempt	Link
🟢 Done	`361bcb3`		1	Explore Profiling Data
🟢 Done	`126b11a`		1	Explore Profiling Data

Powered by Polar Signals Cloud

github-actions · 2026-07-20T20:49:27Z

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +2.2%
Engines: DataFusion Likely regression (+13.0%, medium confidence) · DuckDB No clear signal (-7.5%, medium confidence)
Vortex (geomean): 0.937x ➖
Parquet (geomean): 0.927x ➖
Shifts: Parquet (control) -7.3% · Median polish -5.4%

How to read Verdict and Engines

Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

datafusion / vortex-file-compressed (0.986x ➖, 0↑ 0↓)

name	PR `6f487e2` (ns)	base `037b1a8` (ns)	ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed	9742895	10016685	0.97
vortex_q01/datafusion:vortex-file-compressed	6240138	6241940	1.00

datafusion / parquet (0.872x ✅, 2↑ 0↓)

name	PR `6f487e2` (ns)	base `037b1a8` (ns)	ratio (PR/base)
vortex_q00/datafusion:parquet 🚀	19331045	21628324	0.89
vortex_q01/datafusion:parquet 🚀	4455275	5231454	0.85

duckdb / vortex-file-compressed (0.910x ➖, 1↑ 0↓)

name	PR `6f487e2` (ns)	base `037b1a8` (ns)	ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed	9932590	10577988	0.94
vortex_q01/duckdb:vortex-file-compressed 🚀	5809528	6588883	0.88

duckdb / parquet (0.984x ➖, 0↑ 0↓)

name	PR `6f487e2` (ns)	base `037b1a8` (ns)	ratio (PR/base)
vortex_q00/duckdb:parquet	23116006	23594326	0.98
vortex_q01/duckdb:parquet	9437940	9549441	0.99

No file size changes detected.

32 bytes covered short Iceberg-style keys but not reverse-DNS query-engine keys such as `org.apache.spark.sql.parquet.row.metadata` (41 bytes), which Spark writes into every Parquet file. 64 bytes clears these; with 16 keys the postscript key budget stays at 1 KiB. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: mprammer <martin@spiraldb.com>

The boundary round-trip test builds the largest possible metadata postscript (max segments, one max-length key); assert its serialized size stays within MAX_POSTSCRIPT_SIZE so the footer is always covered by the initial tail read. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: mprammer <martin@spiraldb.com>

Build the true worst case (every key at the max length) and assert the total metadata size alongside the total postscript size, so all keys are exercised at the boundary rather than just one. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: mprammer <martin@spiraldb.com>

Signed-off-by: mprammer <martin@spiraldb.com> # Conflicts: # vortex-file/src/footer/postscript.rs # vortex-file/src/footer/segment.rs # vortex-file/src/open.rs

…nks, Python goldens The develop merge brought in the file-level metadata tests (#8740), which write with bare sessions that enable no editions and now fail validation. Enable the crate's test edition in those sessions, matching the other vortex-file unit tests. Also re-qualify the vortex::editions module-doc links (module docs resolve at the lib.rs `pub mod` site) and update the Python doctest golden file sizes for the edition-derived footer context. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CK2nCuXnyd2g3mNHQC8Lz2

…nks, Python goldens The develop merge brought in the file-level metadata tests (#8740), which write with bare sessions that enable no editions and now fail validation. Enable the crate's test edition in those sessions, matching the other vortex-file unit tests. Also re-qualify the vortex::editions module-doc links (module docs resolve at the lib.rs `pub mod` site) and update the Python doctest golden file sizes for the edition-derived footer context. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CK2nCuXnyd2g3mNHQC8Lz2

mprammer requested review from AdamGS, gatesn and robert3005 July 13, 2026 16:10

robert3005 reviewed Jul 14, 2026

View reviewed changes

Comment thread vortex-flatbuffers/flatbuffers/vortex-file/footer.fbs

mprammer added the changelog/feature A new feature label Jul 15, 2026

mprammer requested a review from robert3005 July 15, 2026 14:38

mprammer and others added 2 commits July 20, 2026 16:13

Merge remote-tracking branch 'origin/develop' into mp/file-metadata

126b11a

Signed-off-by: mprammer <martin@spiraldb.com> # Conflicts: # vortex-file/src/footer/deserializer.rs # vortex-file/src/footer/mod.rs

mprammer marked this pull request as ready for review July 20, 2026 20:51

robert3005 reviewed Jul 21, 2026

View reviewed changes

Comment thread vortex-file/src/open.rs

robert3005 approved these changes Jul 22, 2026

View reviewed changes

robert3005 reviewed Jul 22, 2026

View reviewed changes

Comment thread vortex-file/src/footer/postscript.rs

mprammer and others added 3 commits July 22, 2026 08:48

Merge remote-tracking branch 'origin/develop' into mp/file-metadata

6f487e2

Signed-off-by: mprammer <martin@spiraldb.com> # Conflicts: # vortex-file/src/footer/postscript.rs # vortex-file/src/footer/segment.rs # vortex-file/src/open.rs

robert3005 merged commit c0d1a4e into develop Jul 22, 2026
72 checks passed

robert3005 deleted the mp/file-metadata branch July 22, 2026 15:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add file-level user metadata - #8740

Add file-level user metadata#8740
robert3005 merged 8 commits into
developfrom
mp/file-metadata

mprammer commented Jul 13, 2026

Uh oh!

codspeed-hq Bot commented Jul 13, 2026 •

edited

Loading

Uh oh!

Uh oh!

robert3005 commented Jul 20, 2026

Uh oh!

github-actions Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mprammer commented Jul 13, 2026

Uh oh!

codspeed-hq Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

Uh oh!

robert3005 commented Jul 20, 2026

Uh oh!

github-actions Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Polar Signals Profiling Results

Latest Run

Uh oh!

github-actions Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks: Vortex queries 📖

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codspeed-hq Bot commented Jul 13, 2026 •

edited

Loading

github-actions Bot commented Jul 20, 2026 •

edited

Loading

github-actions Bot commented Jul 20, 2026 •

edited

Loading